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ABSTRACT 

The use of meta-analysis in industrial and 
organizational psychology has become quite common. Unfortunately, the 
understanding and research necessary to ensure appropriate 
application of the technique have not been as Widespread. As part of 
a larger study, a content analysis of meta-analyses from the 
industrial and organizational psychological literature was conducted 
to document the sample sizes— (sub i) , number of studies (K) , 
reliabilities: r(sub xx), r(sub yy); and types of meta-analyses 
occurring in the literature. This content analysis was used to 
identify the typical values associated with data sets found in 
meta-analyses conducted in industrial and organizational psychology. 
Published articles occurring from January 1980 to June 1986 in seven 
journals were included. Meta-analytic papers presented at the 1984 
and 1985 annual meetings of the American Psychological Association 
and the Academy of Management were also reviewed. A total of 81 
meta-analytic studies was found; 19 were selection validation and 62 
were non-validation studies. Results of the content analysis included 
the average reported reliabilities (R(sub xx) =0.79, and R(sub yy) = 
0.71) and the median sample size per correlation— n (sub i) = 102. 
Together, these two findings indicate that the literature is full of 
studies having low power, and that based on other research (J. M. 
Cornwell, 1987; E. R. Kemery, K. W. Mossholder, and L. Roth, 1987; 
and R. T. Ladd and J. M. Cornwell, 1986), such data may lead to 
biased estimates when used in a meta-analysis. Four tables and two 
graphs present study data. (Author/SLD) 
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ABSTRACT 



Content Analysis of Meta-Analytic Studies from I/O Psychology 

The use of Meta-Analysis in industrial and organizational psychology has become 
quite common. Unfortunately, the understanding and research necessary to ensure 
appropriate application of the technique hasnot been as widespread. Aspart ofaiarger 
study a content analysis of meta-analyses from the industrial and organizational 
psychological literature was conducted to document the sample sizes (iij), number of 
studies (^. reliabilities (rxx, ryy), and types of meta-analyses occurring in the literature, 
this content analysis was used to identify the typical values associated with data sets 
found in meta-analyses conducted in industrial and organizational psychology. Results 
of the contentfanalysis included the average reported reliabilities (R xx = .79, R yy = .71) 
and the median sample size per correlation (n j = 102) Together, these two findings 
indicate that the literature is full of studies having low power and that based on other 
research (Cornwell, 1987; Kemery, Mossholder, & Roth, 1987; Ladd & Cornwell, 
1986) such data may lead to biased estimates when used in a meta-analysis. 
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Content Analysis of Meta-Analytic ^.Smjdies Jrom l/Q Psychology. 

The use of Meta-Analysis in industrial and organizational psychology has become 
quite common (Loschehkohl, Faust, Lowenberg, & Conrad, 1984; Schmidt, 1984). 
Unfortunately, the understanding and research necessary to ensure appropriate 
application of the technique has not been as widespread (Kraemer, 1983). Evidence 
suggests that using meta-analytic techniques with data sets having small n's and low 
reliabilities will result in biased estimates (Cornwell, 1987; Kemery, Mossholder, & 
Roth, 1987; Ladd& Cornwell, 1986). 

Method 

As part of a larger study a content analysis of meta-analyses from the industrial 
and organizational psychological literature was conducted to determine the types of data 
being included in meta-analytic studies. This review included published articles 
occurring from January, 1980 until June, 1986 from the following journals: 

1. Psychological Bulletin 

2. Journal of Applied Psychology 

3. Personnel Psychology 

4. American Psychologist 

5 . The Academy of Management Review 

6. Academy of Management Journal 

7. Organizational Behavior and Human Decision Processes 
(formerly Organizational Behavior and Human Performance ) 

Meta-analytic papers presented at the 1984 and 1985 annual meetings of the American 
Psychological Association and the Academy of Management were also reviewed. 

The content analysis of the characteristics of the discovered studies reporting meta- 
analytic results provided information about the sample sizes (n t ), the reliabilities (r xx 
and r yy ), the number of correlations included in a meta-analysis (K), and the meta- 
analytic procedure used (e.g., Glass, McGaw, & Smith, 1981; Hedges, & Olkin, 1985; 
and Hunter, Schmidt, & Jackson, 1982). The abstracts of every article in the seven 



joumak-were jead_to jdentify-meta-analytic studies; Th* programs of the annual 
meetings wjereje^ These 
papers were solicited from their authors through letters mailed before and after the annual 
meetings. 

" - Only-articles"ahd"pa^ " 
analytic results were included in this review. All purely methodological and theoretical 
papers with no results.from a meta-analysis were excluded. No attempt was made to 
remove non-industrial/organizational meta-analyses from the review if the analysis came 
from the above mentioned sources. Selection validation.meta-analyses were grouped 
separately from non-validation meta-analyses to permit comparing the the types of data 
sets used in each group. The author believed that validation meta-analyses would tend to 
have more correlations per meta-analysis (i.e., K) and larger sample sizes per correlation 
(i.e., h j). 

Results 

A total of 81 meta-analytic studies was discovered in the review of the seven 
journals and two years of convention presentations. Table 1 presents the frequency of 
meta-analytic studies by source and whether or not the content concerned selection 
validation. Almost one-fourth of the 81 studies were meta-analyses of selection 
validation studies. 



Insert Table 1 about here 



The frequency of the use of the various meta-analytic procedures was determined 
by analyzing the citations associated with the meta-analyses. The procedure most often 
used is the one advocated by Hunter et al..(1982). It was used with allbut one of the 
validation mita-analytic studies and almost half of the non-validation studies. Table 2 



presents-the~breakdown of-mcta-analytic-proccdure and-contcnt Because-three~of th 
-.articles.reportedmore.than.one^ 

those in Tabic 1. Also, because the procedures have appeared in many articles and 
books, the citation is not necessarily the actual one used by the authors of the meta- 
analyticlitudy: * ~~ ~ 



Insert Table 2 about here 



The content analysis of the meta-analyses found in the literature permitted the 
construction of distributions of the discovered sample sizes (nj) and the discovered 
number of correlations used in a single meta-analytic calculation (K) for both validity and 
non-validity studies. Table 3 gives the descriptive statistics for the ni and K 
distributions. While the means and variances of the nonvalidity and validity meta- 
analyses q>pear to differ greatly, in actuality the shapes of the ni and K distributions of 
the two groups of studies are very similar. Extreme values, as measured by positive 
skewness values in Table 3, contribute to the large differences between the means and 
variances. 



Insert Table 3 about here 



The shapes of the truncated combined distributions for nj and K are shown in 
Figures 1 and 2, respectively. The sample size distributions were truncated so that n x £ 
500 for purposes of graphically presenting the distribution. This resulted in the loss of 
less than nine percent of the observations for the combined distribution which were 
asymptotically distributed near zero percent. Similarly, the distributions of K's were 



- -truncated-at-K-^-l 20-^— This-truncatioh-resulted-in- a-loss-of~less-than-six-percent-of-the 
— observed-ICs.. - 



Insert Figures 1 and 2 about here 



The average reliabilities for the predictor and the criterion (i.e., independent and 
dependent variables) for each of the meta-analytic studies were also abstracted. Twelve 
estimates for the predictor reliability (r xx ) and fourteen estimates for the criterion 
reliability (ryy) were discovered. Table ^presents the mean reliabilities by content and 

the number of individual estimates contributing to each mean. The mean reliabilities for 
the combined studies were r xx = .79 and StL = .71. 



Insert Table 4 about here 

In the majority of the meta-analyses the authors were either unable or unwilling to 
publish information concerning the reliabilities associated with the effect sizes under 
investigation. Of the 19 selection validation studies less than half provided any reliability 
information. All eleven reliabilities listed in Table 4 for validation studies came from 
only eight of the 19 meta-analytic studies. The nonvalidity studies were even less 
informative when it came to reporting reliabilities. Only seven of the 62,nonvalidity 
meta-analytic studies contained any reliability information. In some cases the authors 
reported that the reliabilities were not available from the original studies. In other cases 
such information was available but the authors did not report the estimates. Some 
authors did not report the reliabilities even though they used the estimates in correcting 
for attentuation. 



, Discussion ____ 

Th e typical conditions fo und i n meta-analytic studies from the industrial and 
organizational psychological area were reported in the results. Not surprisingly the most 
often used procedure amcng the meta-analyses reviewed was that of Hunter et al. 
(1982); r-This-procedure^was-developed-by Hunter and his associates-to reconcile^ 
conflicting validation study results within the domain of industrial and organizational 
psychology* It is no wonder that it has predominance in this literature and is virtually the 
only procedure used with validity correlation coefficients. 

A surprising finding is the similarities between characteristics of validity and 
nonvalidity meta-analytic studies. The distributions of n i and K for each type of study 

are remarkedly similar. The median sample size per correlation for the validity 
distribution (n mcdian = 1 16) and the nonvalidity distribution (n median = 99>5) do not 

differ greatly. The two types of studies are also similar in the median number of 
correlations analyzed (i.e., K mcdi a n = 14 for validation versus K median = 11 for 
nonvalidation). The small differences that do exisi are in the direction that is supportive 
of the author's conjecture, that is, the meta-analyses of validity coefficients would have 
larger n (s and have more studies to accumulate. However, these differences are trivial. 
Another similarity between the validation and nonvalidation studies is their average 
reliabilities. Because only fifteen of the 81 meta-analyses that were reviewed reported 
reliabilities, the results may not generalize to the other studies. However, the predictor 
mean reliabilities (r^ differ only by .01 and the criterion mean reliabilities (ryy) differ by 
.052 for the validation and nonvalidation meta-analyses, with the validation studies 
having the higher average reliabilities for both predictor and criterion. 

The characteristics of the validation and nonvalidation studies reveal much about 
the research that has been occurring in industrial and organizational psychology. The, 
median sample size per correlation as represented in the combined meta-analytic studies: 



8 



is 102. If we use this value as representative of the sample size typically found in 
industrial and organizational psychology we can determine what the typical power is by 
making a few reasonable, additional assumptions. Using^Cohen's (197-7) .power 
analysis arid assuming n j = 100, a = ;05, p = .25, and perfectly reliable data,/the 
calculated theoretical PQWer to reject the null hy pothesis is sli ghtly more than 50%. Thus 
almost half of the studies represented in the content analysis would have a Type II error 
of approximately 50% assuming they used a = .05, and the true effect size was p = .25 
and "perfect" data sets with no unreliability or range restriction. 

The other major problem uiat comes to light is the low mean reliabilities of the 
variables being used in correlational and experimental research. Average reliabilities of 
.79 and .71 definitely mean that the estimated effect sizes and therefore the power of the 
research represented is attenuated. Obviously, to the extent that these findings are 
representative, small sample research with its associated problems of low power fill the 
industrial and organizational psychological literature. 

The greatest problem is in using the data in meta-analytic studies. There has been 
ample Mdn«i Carlo simulation research conducted on meta-analytic techniques to indicate 
that use of effect sizes based on small samples and unreliable data (the typical data found 
in this study) will produce biased and unreliable estimates of population effect sizes and 
variances (Cornwell, 1987; Kemery, Mpssholder, & Roth, 1987; Ladd & Cornwell, 
1986). These findings cast doubt on the validity of the meta-analyses that were reviewed 
in this study. 

In summary, an abundance of n j's smaller than 100 and;mean reliabilities.less 
than .80 sums up the condition of research within industrial arid organizational 
psychology as represented by the studies included in the 81 meta-analyses reviewed: 
These studies were also negligent in reporting information concerning reliabilities and the 
degree of range restriction present with the sample used. While several authors 
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bemoaned the lack of such information in the original studies, other authors of meta- 



analytic studies had access to but did not report such information. While it may be 



understandable that the original researcher did not include reliabilities because of 
ignorance, it is certainly not the case with the author of a meta-analytic study who should 
appreciate the importance of such information. Finally, the actual validity of the mcta- 
analyses reviewed may be questioned based upomthe quality of<the datasets typically 
used (i.e., small sample sizes, unreliable data). 
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Table 1 



Frequenc y of Meta-Analytic Studies by Literature Source and by Content 



Industrial/Organizational Psychological Validation Nonvalidation Total 



UpratUi j Sources Reviewed* Studies Studies 

1. Psychological Bulletin 1 24 25 

2. Journal of Applied Psychology 7 10 17 

3. Personnel Psychology 6 2 8 

4. American Psychologist 0 3 3 

5. Academy of Management Review 0 5 5 

6. Academy of Management Journal 0 4 4 

7 . Organizational Behavior and Human 

Decision Processes (formerly Organizational 

Behavior and Human Performance ) 0 2 2 

8. 1984 and 1985 APA Presentations 4 6 10 

9. 1984 and 1985 Academy of Management 

Presentations l 6 7 

Totals 19 62 81 



Meta-analytic journal articles published between January, 1980 and June, 1986 are 
included. A listing of these references is available from the author. 



ii 



FrcquctKy.of.hfeUrAnalytic.Stuclics.by^ 



Meta-analytic Procedure Used (indicated Validation Konvalidation 



by the citation given for the procedure) Studies Studies 

1. Hunter etal. (1982) 18 30 

2. Glass et al. (1981) 0 13 

3. Hedges etal. (1985) 0 5 

4. Rosenthal (1984 ) 0 8 

5. Miscellaneous (Z-test, summing 1 9 

probability levels, voting method) 

Totals 19 65 
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Table 3 

Descriptive Statistics for the Distributions of Sample Size (nj) and Number of 
Correlations included in a Meta-Analysis Calculation (K) by Content 



Sample Size Number of Correlations 

(Hi) Used(K) 



Descriptive 

Cfratictir* 

OlaUSUC 


Selection Non 
Validation validation 


Combined 


Selection Non 
Validation validation 


Combined 


Number of 














Observations 3 


695 


1802 


2497 


323 


302 


625 


Mean 


301.39 


275.61 


282.78 


55.2 


17.5 


37.01 


Median 


116 


99.5 


102 


14 


11 


12 


Mode 


100 


73 


73 


10 


6 


6 


Standard Deviation 


576.05 


1680.69 


1459.66 


155 


26.6 


114.43 


Variance 


331,837 


2,824,726 


2,130,593 


24,024 


709.4 


13,093.7 


Skewness 


5.27 


22.7 


25.07 


9.00 


7.50 


11.99 


Kurtosis 


34.67 


579.54 


734.91 


109.55 


74.73 


198.67 


Minimum 


7 


10 


7 


2 


2 


2 


Maximum 


5,635 


45,222 


45,222 


2,162 


330 


2,162 



a Number of observations refers to the total number of correlation sample sizes (/if) 
actually reported in all the articles reviewed or, for the second column, to the total 
numberoHndividually-calculated-meta-analyses reported-for which K could- be 
determined (many articles reported calculating multiple meta-analyses using subsets of 
the correlations). 
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Table 4 

Mean Reliabilities of Predictor (F^) and Criterion (Fyy) and the Number of Observations 
for Each by Content of Study 



Content • r. 



xx n r, 



yy 



Nonvalidation Studies .786 7 .692 8 

Selection Validation Studies .796 5 .744 6 

Combined Studies .790 12 .714 14 
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Mode = 73 
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Figure 1 The truncated distribution of sample sizes (nj) per correlation coefficient 
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Figure 2 The truncated distribution of K, the number of correlation coefficients included 
per meta-analysis. 
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